What we do (not) know about the mechanisms underlying adaptive speech perception: A computational framework and review.

Speech from unfamiliar talkers can be difficult to comprehend initially. These difficulties tend to dissipate with exposure, sometimes within minutes or less. Adaptivity in response to unfamiliar input is now considered a fundamental property of speech perception, and research over the past two decades has made substantial progress in identifying its characteristics. The mechanisms underlying adaptive speech perception, however, remain unknown. Past work has attributed facilitatory effects of exposure to any one of three qualitatively different hypothesized mechanisms: (1) low-level, pre-linguistic, signal normalization, (2) changes in/selection of linguistic representations, or (3) changes in post-perceptual decision-making. Direct comparisons of these hypotheses, or combinations thereof, have been lacking. We describe a general computational framework for adaptive speech perception (ASP) that-for the first time-implements all three mechanisms. We demonstrate how the framework can be used to derive predictions for experiments on perception from the acoustic properties of the stimuli. Using this approach, we find that-at the level of data analysis presently employed by most studies in the field-the signature results of influential experimental paradigms do not distinguish between the three mechanisms. This highlights the need for a change in research practices, so that future experiments provide more informative results. We recommend specific changes to experimental paradigms and data analysis. All data and code for this study are shared via OSF, including the R markdown document that this article is generated from, and an R library that implements the models we present.

Evaluating normalization accounts against the dense vowel space of Central Swedish.

Talkers vary in the phonetic realization of their vowels. One influential hypothesis holds that listeners overcome this inter-talker variability through pre-linguistic auditory mechanisms that normalize the acoustic or phonetic cues that form the input to speech recognition. Dozens of competing normalization accounts exist-including both accounts specific to vowel perception and general purpose accounts that can be applied to any type of cue. We add to the cross-linguistic literature on this matter by comparing normalization accounts against a new phonetically annotated vowel database of Swedish, a language with a particularly dense vowel inventory of 21 vowels differing in quality and quantity. We evaluate normalization accounts on how they differ in predicted consequences for perception. The results indicate that the best performing accounts either center or standardize formants by talker. The study also suggests that general purpose accounts perform as well as vowel-specific accounts, and that vowel normalization operates in both temporal and spectral domains.

Using Rational Models to Interpret the Results of Experiments on Accent Adaptation.

Exposure to unfamiliar non-native speech tends to improve comprehension. One hypothesis holds that listeners adapt to non-native-accented speech through distributional learning-by inferring the statistics of the talker's phonetic cues. Models based on this hypothesis provide a good fit to incremental changes after exposure to atypical native speech. These models have, however, not previously been applied to non-native accents, which typically differ from native speech in many dimensions. Motivated by a seeming failure to replicate a well-replicated finding from accent adaptation, we use ideal observers to test whether our results can be understood solely based on the statistics of the relevant cue distributions in the native- and non-native-accented speech. The simple computational model we use for this purpose can be used predictively by other researchers working on similar questions. All code and data are shared.

A Rational Model of Incremental Argument Interpretation: The Comprehension of Swedish Transitive Clauses.

A central component of sentence understanding is verb-argument interpretation, determining how the referents in the sentence are related to the events or states expressed by the verb. Previous work has found that comprehenders change their argument interpretations incrementally as the sentence unfolds, based on morphosyntactic (e.g., case, agreement), lexico-semantic (e.g., animacy, verb-argument fit), and discourse cues (e.g., givenness). However, it is still unknown whether these cues have a privileged role in language processing, or whether their effects on argument interpretation originate in implicit expectations based on the joint distribution of these cues with argument assignments experienced in previous language input. We compare the former, linguistic account against the latter, expectation-based account, using data from production and comprehension of transitive clauses in Swedish. Based on a large corpus of Swedish, we develop a rational (Bayesian) model of incremental argument interpretation. This model predicts the processing difficulty experienced at different points in the sentence as a function of the Bayesian surprise associated with changes in expectations over possible argument interpretations. We then test the model against reading times from a self-paced reading experiment on Swedish. We find Bayesian surprise to be a significant predictor of reading times, complementing effects of word surprisal. Bayesian surprise also captures the qualitative effects of morpho-syntactic and lexico-semantic cues. Additional model comparisons find that it-with a single degree of freedom-captures much, if not all, of the effects associated with these cues. This suggests that the effects of form- and meaning-based cues to argument interpretation are mediated through expectation-based processing.

Cross-talker generalization in the perception of nonnative speech: A large-scale replication.

Speech perception depends on the ability to generalize previously experienced input effectively across talkers. How such cross-talker generalization is achieved has remained an open question. In a seminal study, Bradlow & Bent (2008, henceforth BB08) found that exposure to just 5 min of accented speech can elicit improved recognition that generalizes to an unfamiliar talker of the same accent (N = 70 participants). Cross-talker generalization was, however, only observed after exposure to multiple talkers of the accent, not after exposure to a single accented talker. This contrast between single- and multitalker exposure has been highly influential beyond research on speech perception, suggesting a critical role of exposure variability in learning and generalization. We assess the replicability of BB08's findings in two large-scale perception experiments (total N = 640) including 20 unique combinations of exposure and test talkers. Like BB08, we find robust evidence for cross-talker generalization after multitalker exposure. Unlike BB08, we also find evidence for generalization after single-talker exposure. The degree of cross-talker generalization depends on the specific combination of exposure and test talker. This and other recent findings suggest that exposure to cross-talker variability is not necessary for cross-talker generalization. Variability during exposure might affect generalization only indirectly, mediated through the informativeness of exposure about subsequent speech during test: Similarity-based inferences can explain both the original BB08 and the present findings. We present Bayesian data analysis, including Bayesian meta-analyses and replication tests for generalized linear mixed models. All data, stimuli, and reproducible literate (R markdown) code are shared via OSF. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

(Early) context effects on event-related potentials over natural inputs.

Language understanding requires the integration of the input with preceding context. Event-related potentials (ERPs) have contributed significantly to our understanding of what contextual information is accessed and when. Much of this research has, however, been limited to experimenter-designed stimuli with highly atypical lexical and context statistics. This raises questions about the extent to which previous findings generalize to everyday language processing of natural stimuli with typical linguistic statistics. We ask whether context can affect ERPs over natural stimuli early, before the N400 time window. We re-analyzed a data set of ERPs over ~700 visually presented content words in sentences from English novels. To increase power, we employed linear mixed effects regression simultaneously modeling random variance by subject and by item. To reduce concerns about Type I error inflation common to any type of time series analysis, we introduced a simple approach to model and discount auto-correlations at multiple, empirically determined, time lags. We compared this approach to Bonferroni correction. Planned follow-up analyses used Generalized Additive Mixture Models to assess the linearity of contextual effects, including lexical surprisal, found within the N400 time window. We found that contextual information affects ERPs in both early (~200ms after word onset) and late (N400) time windows, supporting a cascading, interactive account of lexical access.

Comparing non-native and native speech: Are L2 productions more variable?

Foreign-accented speech of second language learners is often difficult to understand for native listeners of that language. Part of this difficulty has been hypothesized to be caused by increased within-category variability of non-native speech. However, until recently, there have been few direct tests for this hypothesis. The realization of vowels and word-final stops in productions of native-English L1 speakers and native-Mandarin speakers of L2 English is compared. With the largest sample size to date, it is shown that at least proficient non-native speakers exhibit little or no difference in category variability compared to native speakers. This is shown while correcting for the effects of phonetic context. The same non-native speakers show substantial deviations from native speech in the central tendencies (means) of categories, as well as in the correlations among cues they produce. This relativizes a common and a priori plausible assumption that competition between first and second language representations necessarily leads to increased variability-or, equivalently, decreased precision, consistency, and stability-of non-native speech. Instead, effects of non-nativeness on category variability are category- and cue-specific.

Production efficiency can cause grammatical change: Learners deviate from the input to better balance efficiency against robust message transmission.

The idea that human languages have properties suitable for efficient communication has permeated linguistic theorizing. Indirect correlational support for this idea has come from cross-linguistic synchronic and diachronic data. However, direct causal tests have been lacking. We directly test whether biases operating during language learning can cause learners to deviate from the input they receive towards output languages that better balance production efficiency against robust message transmission. We employ miniature language learning experiments to address this question for a well-documented cross-linguistic correlation between constituent order flexibility and the presence of case marking in a language. Participants were exposed to novel miniature languages that had optional case marking and either fixed or flexible constituent order. Between participants, we manipulated the amount of time and effort associated with the production of case marking. We find that learners introduced the cross-linguistically observed trade-off between case marking and constituent order flexibility into their output languages. Critically, learners only did so when case-marked nouns required additional effort compared to non-case-marked nouns. Thus, the present study suggests that even abstract grammatical properties of languages can be shaped by a balance between production efficiency and robust message transmission.

Big data suggest strong constraints of linguistic similarity on adult language learning.

When adults learn new languages, their speech often remains noticeably non-native even after years of exposure. These non-native variants ('accents') can have far-reaching socio-economic consequences for learners. Many factors have been found to contribute to a learners' proficiency in the new language. Here we examine a factor that is outside of the control of the learner, linguistic similarities between the learner's native language (L1) and the new language (Ln). We analyze the (open access) speaking proficiencies of about 50,000 Ln learners of Dutch with 62 diverse L1s. We find that a learner's L1 accounts for 9-22% of the variance in Ln speaking proficiency. This corresponds to 28-69% of the variance explained by a model with controls for other factors known to affect language learning, such as education, age of acquisition and length of exposure. We also find that almost 80% of the effect of L1 can be explained by combining measures of phonological, morphological, and lexical similarity between the L1 and the Ln. These results highlight the constraints that a learner's native language imposes on language learning, and inform theories of L1-to-Ln transfer during Ln learning and use. As predicted by some proposals, we also find that L1-Ln phonological similarity is better captured when subcategorical properties (phonological features) are considered in the calculation of phonological similarities.

Talker-specific pronunciation or speech error? Discounting (or not) atypical pronunciations during speech perception.

Perceptual recalibration allows listeners to adapt to talker-specific pronunciations, such as atypical realizations of specific sounds. Such recalibration can facilitate robust speech recognition. However, indiscriminate recalibration following any atypically pronounced words also risks interpreting pronunciations as characteristic of a talker that are in reality because of incidental, short-lived factors (such as a speech error). We investigate whether the mechanisms underlying perceptual recalibration involve inferences about the causes for unexpected pronunciations. In 5 experiments, we ask whether perceptual recalibration is blocked if the atypical pronunciations of an unfamiliar talker can also be attributed to other incidental causes. We investigated 3 type of incidental causes for atypical pronunciations: the talker is intoxicated, the talker speaks unusually fast, or the atypical pronunciations occur only in the context of tongue twisters. In all 5 experiments, we find robust evidence for perceptual recalibration, but little evidence that the presence of incidental causes block perceptual recalibration. We discuss these results in light of other recent findings that incidental causes can block perceptual recalibration. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Dynamic re-weighting of acoustic and contextual cues in spoken word recognition.

Listeners integrate acoustic and contextual cues during word recognition. However, experiments investigating this integration disrupt natural cue correlations. It was investigated whether changes in correlational structure affect listeners' relative cue weightings. Two groups of participants engaged in a word recognition task. In one group, acoustic (voice onset time) and contextual (lexical bias) cues followed natural correlations; in the other, cues were uncorrelated. When cues were correlated, cue weights were stable throughout the experiment; when cues were uncorrelated, contextual cues were down-weighted. Listeners thus can re-weight cues based on their statistical structure. Studies failing to account for re-weighting risk over/under-estimating cue importance.

Maintaining information about speech input during accent adaptation.

Speech understanding can be thought of as inferring progressively more abstract representations from a rapidly unfolding signal. One common view of this process holds that lower-level information is discarded as soon as higher-level units have been inferred. However, there is evidence that subcategorical information about speech percepts is not immediately discarded, but is maintained past word boundaries and integrated with subsequent input. Previous evidence for such subcategorical information maintenance has come from paradigms that lack many of the demands typical to everyday language use. We ask whether information maintenance is also possible under more typical constraints, and in particular whether it can facilitate accent adaptation. In a web-based paradigm, participants listened to isolated foreign-accented words in one of three conditions: subtitles were displayed concurrently with the speech, after speech offset, or not displayed at all. The delays between speech offset and subtitle presentation were manipulated. In a subsequent test phase, participants then transcribed novel words in the same accent without the aid of subtitles. We find that subtitles facilitate accent adaptation, even when displayed with a 6 second delay. Listeners thus maintained subcategorical information for sufficiently long to allow it to benefit adaptation. We close by discussing what type of information listeners maintain-subcategorical phonetic information, or just uncertainty about speech categories.

Rapid adaptation to foreign-accented speech and its transfer to an unfamiliar talker.

How fast can listeners adapt to unfamiliar foreign accents? Clarke and Garrett [J. Acoust. Soc. Am. 116, 3647-3658 (2004)] (CG04) reported that native-English listeners adapted to foreign-accented English within a minute, demonstrating improved processing of spoken words. In two web-based experiments that closely follow the design of CG04, the effects of rapid accent adaptation are examined and its generalization is explored across talkers. Experiment 1 replicated the core finding of CG04 that initial perceptual difficulty with foreign-accented speech can be attenuated rapidly by a brief period of exposure to an accented talker. Importantly, listeners showed both faster (replicating CG04) and more accurate (extending CG04) comprehension of this talker. Experiment 2 revealed evidence that such adaptation transferred to a different talker of a same accent. These results highlight the rapidity of short-term accent adaptation and raise new questions about the underlying mechanism. It is suggested that the web-based paradigm provides a useful tool for investigations in speech adaptation.

Inferring causes during speech perception.

One of the central challenges in speech perception is the lack of invariance: talkers differ in how they map words onto the speech signal. Previous work has shown that one mechanism by which listeners overcome this variability is adaptation. However, talkers differ in how they pronounce words for a number of reasons, ranging from more permanent, characteristic factors such as having a foreign accent, to more temporary, incidental factors, such as speaking with a pen in the mouth. One challenge for listeners is that the true cause underlying atypical pronunciations is never directly known, and instead must be inferred from (often causally ambiguous) evidence. In three experiments, we investigate whether these inferences underlie speech perception, and how the speech perception system deals with uncertainty about competing causes for atypical pronunciations. We find that adaptation to atypical pronunciations is affected by whether the atypical pronunciations are seen as characteristic or incidental. Furthermore, we find that listeners are able to maintain information about previous causally ambiguous pronunciations that they experience, and use this previously experienced evidence to drive their adaptation after additional evidence has disambiguated the cause. Our findings revise previous proposals that causally ambiguous evidence is ignored during speech adaptation.

Corrigendum: What the Heck Is Salience? How Predictive Language Processing Contributes to Sociolinguistic Perception.

[This corrects the article on p. 1115 in vol. 7, PMID: 27536257.].

Balancing Effort and Information Transmission During Language Acquisition: Evidence From Word Order and Case Marking.

Across languages of the world, some grammatical patterns have been argued to be more common than expected by chance. These are sometimes referred to as (statistical) language universals. One such universal is the correlation between constituent order freedom and the presence of a case system in a language. Here, we explore whether this correlation can be explained by a bias to balance production effort and informativity of cues to grammatical function. Two groups of learners were presented with miniature artificial languages containing optional case marking and either flexible or fixed constituent order. Learners of the flexible order language used case marking significantly more often. This result parallels the typological correlation between constituent order flexibility and the presence of case marking in a language and provides a possible explanation for the historical development of Old English to Modern English, from flexible constituent order with case marking to relatively fixed order without case marking. In addition, learners of the flexible order language conditioned case marking on constituent order, using more case marking with the cross-linguistically less frequent order, again mirroring typological data. These results suggest that some cross-linguistic generalizations originate in functionally motivated biases operating during language learning.

Incremental implicit learning of bundles of statistical patterns.

Forming an accurate representation of a task environment often takes place incrementally as the information relevant to learning the representation only unfolds over time. This incremental nature of learning poses an important problem: it is usually unclear whether a sequence of stimuli consists of only a single pattern, or multiple patterns that are spliced together. In the former case, the learner can directly use each observed stimulus to continuously revise its representation of the task environment. In the latter case, however, the learner must first parse the sequence of stimuli into different bundles, so as to not conflate the multiple patterns. We created a video-game statistical learning paradigm and investigated (1) whether learners without prior knowledge of the existence of multiple "stimulus bundles" - subsequences of stimuli that define locally coherent statistical patterns - could detect their presence in the input and (2) whether learners are capable of constructing a rich representation that encodes the various statistical patterns associated with bundles. By comparing human learning behavior to the predictions of three computational models, we find evidence that learners can handle both tasks successfully. In addition, we discuss the underlying reasons for why the learning of stimulus bundles occurs even when such behavior may seem irrational.

Now or later: Perceptual data are not immediately forgotten during language processing.

Christiansen & Chater (C&C) propose that language comprehenders must immediately compress perceptual data by "chunking" them into higher-level categories. Effective language understanding, however, requires maintaining perceptual information long enough to integrate it with downstream cues. Indeed, recent results suggest comprehenders do this. Although cognitive systems are undoubtedly limited, frameworks that do not take into account the tasks that these systems evolved to solve risk missing important insights.